Skip to content

Test failure summaries in CI#5594

Open
neilmayhew wants to merge 4 commits intomasterfrom
nm/ci-test-failure-summaries
Open

Test failure summaries in CI#5594
neilmayhew wants to merge 4 commits intomasterfrom
nm/ci-test-failure-summaries

Conversation

@neilmayhew
Copy link
Contributor

@neilmayhew neilmayhew commented Feb 26, 2026

Description

Add a small utility that extracts test failure info from log files and uploads them as JSON artifacts. Another utility in the same package then collates the failure artifacts at the end of the whole CI run and publishes a summary to the run web page. This reduces the need for a human to look at the log files, and provides the information necessary for reproducing failures locally (target, pattern and seed).

This summary could also be included in the Slack post from nightly failures, but it would be noisy and the summary is only a single click away anyway.

Checklist

  • Commits in meaningful sequence and with useful messages.
  • Tests added or updated when needed.
  • CHANGELOG.md files updated for packages with externally visible changes.
    NOTE: New section is never added with the code changes. (See RELEASING.md).
  • Versions updated in .cabal and CHANGELOG.md files when necessary, according to the
    versioning process.
  • Version bounds in .cabal files updated when necessary.
    NOTE: If bounds change in a cabal file, that package itself must have a version increase. (See RELEASING.md).
  • Code formatted (use scripts/fourmolize.sh).
  • Cabal files formatted (use scripts/cabal-format.sh).
  • CDDL files are up to date (use scripts/gen-cddl.sh)
  • hie.yaml updated (use scripts/gen-hie.sh).
  • Self-reviewed the diff.

@neilmayhew
Copy link
Contributor Author

neilmayhew commented Feb 26, 2026

This is a draft for now because it contains a deliberate failure to show what the output looks like.

It can't be merged anyway with the failure in it, so I'll mark it ready for review.

@neilmayhew neilmayhew marked this pull request as ready for review February 26, 2026 00:45
@neilmayhew neilmayhew requested a review from a team as a code owner February 26, 2026 00:46
@neilmayhew neilmayhew force-pushed the nm/ci-test-failure-summaries branch 4 times, most recently from 659a5c5 to 0c4e63b Compare February 27, 2026 06:49
Copy link
Contributor

@aniketd aniketd left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think this is going to be super useful. Thanks @neilmayhew !

@neilmayhew neilmayhew force-pushed the nm/ci-test-failure-summaries branch from 0c4e63b to c69deee Compare February 27, 2026 18:34
@neilmayhew neilmayhew force-pushed the nm/ci-test-failure-summaries branch from c69deee to ae152f5 Compare February 28, 2026 00:29
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants